Electronic apparatus for recognition of a user and operation method thereof

ABSTRACT

An electronic apparatus is provided. The electronic apparatus includes a camera configured to obtain a user image by capturing an image of a user, a memory configured to store one or more instructions, and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is further configured to, by executing the one or more instructions, recognize the user from a face region of the user image by using a first recognition model learned based on face information of a plurality of users, extract additional feature information regarding the recognized user from the user image, allow the first recognition model to additionally learn based on the extracted additional feature information, recognize the user from a person region of the user image by using an additionally learned second recognition model, and output a recognition result of the second recognition model.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 15/850,393, filed on Dec. 21, 2017, which claimed the benefit under35 U.S.C. § 119(a) of a Korean patent application filed on Dec. 23, 2016in the Korean Intellectual Property Office and assigned Serial number10-2016-0177884, and of a Korean patent application filed on Oct. 19,2017 in the Korean Intellectual Property Office and assigned Serialnumber 10-2017-0135870, the entire disclosure each of which is herebyincorporated by reference.

TECHNICAL FIELD

The present disclosure relates to electronic apparatuses and operationmethods thereof. More particularly, the present disclosure relates toelectronic apparatuses capable of recognizing a user based on a userimage, and operation methods of the electronic apparatuses.

In addition, the present disclosure relates to an artificialintelligence (AI) system for mimicking functions of a human brain, suchas cognition and judgment, by using a machine learning algorithm, andapplication of the AI system.

BACKGROUND

Facial recognition technology means technology for extracting, from auser's face, various information such as a length or a distance of eachorgan, such as eyes, nose, and mouth, included in the face, andrecognizing the user by analyzing the extracted information.

Facial recognition technology may be used for identification purposesinstead of an identification card, a passport, a credit card, etc., andmay also be used for security purposes such as commuter management, doorpass control, and password replacement. In addition, facial recognitiontechnology may be used for public order purposes such as searching for acriminal suspect and surveillance of a crime-ridden district.

User recognition using facial recognition technology involvesrecognizing a user based on features extracted from a face region of auser image and thus has a problem in that, when a front of the face isnot included in the user image, or a user's face appears small becausean image of the user is captured from a distance, it is difficult torecognize the face.

In addition, an artificial intelligence (AI) system that implementshuman-level intelligence has been recently used in the field of facialrecognition. Unlike an existing rule-based smart system, the AI systemallows a machine to learn by itself, make decisions, and become smarter.As the AI system is used, the AI system has an improved recognition rateand accurately understands a user's preference, and thus, the existingrule-based smart system is gradually being replaced with adeep-learning-based AI system.

AI technology includes machine learning (e.g., deep learning) andelement technologies using machine learning.

Machine learning is an algorithm technique that classifies/learnscharacteristics of input data by itself, and element technologies aretechnologies that simulate a function such as recognition, decisionmaking, etc., of a human brain by using a machine-learning algorithmsuch as deep learning, and include technical fields such as linguisticunderstanding, visual understanding, inference/prediction, knowledgerepresentation, operation control, and so forth.

The AI technology is employed in various fields. For example, linguisticunderstanding is a technique that recognizes, and applies/processeshuman languages/texts, and includes natural language processing, machineinterpretation, a conversation system, question and answer processing,voice recognition/synthesis, and so forth. Visual understanding is atechnique that recognizes and processes an object in the same manner asa human visual system, and includes object recognition, object tracking,image searching, people recognition, scene understanding, spaceunderstanding, image enhancement, etc. Inference/prediction is atechnique that determines information and performs logical inference andprediction based thereon, and includes knowledge/probability-basedinference, optimization prediction, preference-basedplanning/recommendation, and so forth. Knowledge representation is atechnique that automatizes human experience information as knowledgedata, and includes knowledge establishment (datacreation/classification), knowledge management (data utilization), andthe like. Operation control is a technique that controls autonomousdriving of a vehicle and motion of a robot, and includes motion control(navigation, collision, driving), manipulation control (action control),and so forth.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentdisclosure is to provide electronic apparatuses capable of recognizing auser by using appearance information or behavior information of a userin addition to face information of the user, and operation methods ofthe electronic apparatuses.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the present disclosure, an electronicapparatus is provided. The apparatus includes a camera configured toobtain a user image by capturing an image of a user, a memory configuredto store one or more instructions, and a processor configured to executethe one or more instructions stored in the memory, wherein the processoris further configured to, by executing the one or more instructions,recognize the user from a face region of the user image by using a firstrecognition model learned based on face information of a plurality ofusers, extract additional feature information regarding the recognizeduser from the user image, allow the first recognition model toadditionally learn based on the extracted additional featureinformation, recognize the user from a person region of the user imageby using an additionally learned second recognition model, and output arecognition result of the second recognition model.

In accordance with another aspect of the present disclosure, anoperation method of an electronic apparatus is provided. The methodincludes obtaining a user image by capturing an image of a user,recognizing the user from a face region of the user image by using afirst recognition model learned based on face information of a pluralityof users, extracting additional feature information regarding therecognized user from the user image, allowing the first recognitionmodel to additionally learn based on the extracted additional featureinformation of the user, recognizing the user from a person region ofthe user image by using an additionally learned second recognitionmodel, and outputting a recognition result of at least one of the secondrecognition model.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram of a method, performed by an electronic apparatus,of recognizing a user, according to an embodiment of the presentdisclosure;

FIG. 2 is a flowchart of an operation method of an electronic apparatus,according to an embodiment of the present disclosure;

FIG. 3 is a diagram for explaining a method of recognizing a user byusing face information, according to an embodiment of the presentdisclosure;

FIG. 4 is a diagram for explaining a method of extracting additionalfeature information, according to an embodiment of the presentdisclosure;

FIG. 5 is a diagram showing a recognition result of a first recognitionmodel and a recognition result of a second recognition model, accordingto an embodiment of the present disclosure;

FIG. 6 is a flowchart of a method in which an electronic apparatusupdates a second recognition model, according to an embodiment of thepresent disclosure;

FIG. 7 is a flowchart of a method in which an electronic apparatusupdates a second recognition model, according to an embodiment of thepresent disclosure;

FIG. 8 is a diagram of a method in which an electronic apparatusdetermines whether to update a recognition model, according to anembodiment of the present disclosure;

FIG. 9 is a diagram of a method in which an electronic apparatusdetermines whether to update a recognition model, according to anembodiment of the present disclosure;

FIG. 10 is a block diagram of a structure of an electronic apparatus,according to an embodiment of the present disclosure;

FIG. 11 is a block diagram of a processor according to variousembodiments of the present disclosure;

FIG. 12 is a block diagram of a data learning unit according to variousembodiments of the present disclosure;

FIG. 13 is a block diagram of a data recognition unit according tovarious embodiments of the present disclosure;

FIG. 14 is a diagram of an example in which data is learned andrecognized by an electronic apparatus and a server interworking witheach other, according to an embodiment of the present disclosure; and

FIG. 15 is a block diagram of a structure of an electronic apparatus,according to an embodiment of the present disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding, but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purposes only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

The terms used in the present specification will be briefly described,and the present disclosure will be described in detail. As used herein,the term “and/or” includes any and all combinations of one or more ofthe associated listed items. Expressions such as “at least one of,” whenpreceding a list of elements, modify the entire list of elements and donot modify the individual elements of the list.

The terms used in the present disclosure are selected from among commonterms that are currently widely used in consideration of their functionin the present disclosure. However, the terms may be different accordingto an intention of one of ordinary skill in the art, a precedent, or theadvent of new technology. In addition, in particular cases, the termsare discretionally selected by the applicant, and the meaning of thoseterms will be described in detail in the corresponding part of thedetailed description. Therefore, the terms used in the presentdisclosure are not merely designations of the terms, but the terms aredefined based on the meaning of the terms and content throughout thepresent disclosure.

Throughout the present application, when a part “includes” an element,it is to be understood that the part additionally includes otherelements rather than excluding other elements as long as there is noparticular opposing recitation. In addition, the terms such as “ . . .unit”, “-or”, “module”, or the like used in the present applicationindicate a unit which processes at least one function or motion, and theunit may be implemented as hardware or software or by a combination ofhardware and software.

Embodiments will now be described more fully with reference to theaccompanying drawings so that those of ordinary skill in the art maypractice the embodiments without any difficulty. However, the presentembodiments may have different forms and should not be construed asbeing limited to the descriptions set forth herein. In addition, partsin the drawings unrelated to the detailed description are omitted toensure clarity of the present disclosure, and like reference numerals inthe drawings denote like elements throughout.

FIG. 1 is a diagram of a method, performed by an electronic apparatus100, of recognizing a user, according to an embodiment of the presentdisclosure.

Referring to FIG. 1, the electronic apparatus 100 may have variousforms. For example, the electronic apparatus 100 may be implemented asvarious kinds of electronic apparatuses, such as a closed circuittelevision (CCTV), a video phone, a camera, a smart door lock, acellular phone, a smartphone, a laptop computer, a desktop, a tabletpersonal computer (PC), an e-book reader, a digital broadcastingterminal, a personal digital assistant (PDA), a portable multimediaplayer (PMP), a navigation apparatus, a MP3 player, a camcorder, anInternet protocol television (IPTV), a digital television (DTV), awearable device (for example, a smart glass, etc.), etc. However, thepresent disclosure is not limited thereto. In addition, although FIG. 1shows the electronic apparatus 100 that obtains a user image 20 bycapturing an image of a user, the present disclosure is not limitedthereto, and the electronic apparatus 100 according to an embodiment mayreceive a user image captured in an external camera apparatus.

The term “user” in embodiments of the present specification may refer toa person who controls a function or an operation of an electronicapparatus and may include a manager or an installer. Alternatively, theuser may refer to a person recognized by an electronic apparatus.

The electronic apparatus 100 may register face information regarding aplurality of users. For example, the electronic apparatus 100 may obtainuser images 10 regarding a plurality of users intended to be registered.The user images 10 may include an image captured by using an electronicapparatus, an image stored in an electronic apparatus, or an imagereceived from an external apparatus. In addition, the user images 10 maybe images where face regions may be detected, and for example, may beimages including front face parts of users (user A, user B, and user C).

The electronic apparatus 100 may extract face information from the userimages 10 by using a first recognition model 30 and may allow the firstrecognition model 30 to learn by using the extracted face information.The first recognition model 30 may be a model of recognizing a userbased on face information. In addition, the first recognition model 30may be a model based on neural network. For example, a model such asdeep neural network (DNN), recurrent neural network (RNN), orbidirectional recurrent deep neural network (BRDNN) may be used as thefirst recognition model 30, but the present disclosure is not limitedthereto.

The electronic apparatus 100 may obtain the user image 20 by capturingan image of a user intended to be recognized. However, the presentdisclosure is not limited thereto. The electronic apparatus 100 maydetect a face region 21 from the user image 20 by using the firstrecognition model 30, may extract face information of the user from thedetected face region 21, and may recognize which one of the registeredusers is the user based on the extracted face information.

In addition, the electronic apparatus 100 may detect a person region 22from the user image 20. The electronic apparatus 100 may extractadditional feature information 40 such as appearance information orbehavior information of the user from the person region 22. For example,the electronic apparatus 100 may extract the additional featureinformation 40 such as appearance information or behavior information ofthe user by using a feature extraction model, but the present disclosureis not limited thereto.

When user A is recognized in the user image 20 based on faceinformation, the electronic apparatus 100 may match the extractedadditional feature information 40 with user A to generate a secondrecognition model 50 with the first recognition model 30 havingadditionally learned. The second recognition model 50 according to anembodiment may be a model of recognizing a user based on the additionalfeature information 40 in addition to the face information. In addition,the second recognition model 50 may be a model based on neural network.For example, a model such as DNN, RNN, or BRDNN may be used as thesecond recognition model 50, but the present disclosure is not limitedthereto.

In a case of using the second recognition model 50, even when theelectronic apparatus 100 obtains a user image 60 where a face region isnot detected, the electronic apparatus 100 may extract additionalfeature information from the user image 60 and may recognize user Abased on the extracted additional feature information.

The electronic apparatus 100 may determine whether it is necessary toupdate an existing second recognition model, based on a user recognitionresult through the second recognition model. When it is determined thatthe second recognition model is necessary to update, the secondrecognition model may be allowed to additionally learn by using userdata. The user data may include a user image and user information mappedto the user image, or additional feature information extracted from auser image and user information mapped to the additional featureinformation. However, the present disclosure is not limited thereto.

FIG. 2 is a flowchart of an operation method of an electronic apparatus,according to an embodiment of the present disclosure.

Referring to FIG. 2, in operation S210, the electronic apparatus 100 mayobtain a user image intended to be recognized. For example, the userimage may include an image captured in the electronic apparatus 100, animage stored in the electronic apparatus 100, or an image received froman external apparatus. However, the present disclosure is not limitedthereto.

In operation S220, the electronic apparatus 100 may recognize a userfrom a face region of the user image by using a first recognition model.The first recognition model may be a model learned by using faceinformation regarding a plurality of registered users, and may be amodel of recognizing a user based on face information. For example, theelectronic apparatus 100 may detect a face region in a user image, mayextract face information of a user from the detected face region, andmay recognize which one of the registered users is the user based on theextracted face information. A method of recognizing a user based on faceinformation is described below in detail with reference to FIG. 3.

In operation S230, the electronic apparatus 100 may extract additionalfeature information regarding the recognized user. For example, theelectronic apparatus 100 may extract additional feature information fromthe user image obtained in operation S210. The electronic apparatus 100may detect a person region in the user image and may extract additionalfeature information including appearance information or behaviorinformation of the user from the detected person region. A method ofextracting additional feature information is described in detail belowwith reference to FIG. 4.

Alternatively, the electronic apparatus 100 may obtain user imagesregarding the user recognized in operation S220 in real time and mayextract additional feature information from the images obtained in realtime.

In operation S240, the electronic apparatus 100 may allow the firstrecognition model to additionally learn based on the additional featureinformation. For example, the electronic apparatus 100 may allow thefirst recognition model to additionally learn by matching additionalfeature information extracted with respect to user A with user A andthus may generate a second recognition model. The second recognitionmodel may be a model of recognizing a user based on additional featureinformation in addition to face information of the user.

In operation S250, the electronic apparatus 100 may recognize a userfrom a person region of the user image by using an additionally learnedsecond recognition model. For example, even when the electronicapparatus 100 obtains a user image where a face region is not detected,the electronic apparatus 100 may extract additional feature informationfrom a person region of the user image, and may recognize a user basedon the extracted additional feature information.

In operation S260, the electronic apparatus 100 may output a recognitionresult. For example, as a result of recognizing a user, the electronicapparatus 100 may display the recognized user, or may generate an alarmor output a warning message when the recognized user is not a fair user.However, the present disclosure is not limited thereto.

FIG. 3 is a diagram of a method of recognizing a user by using faceinformation, according to an embodiment of the present disclosure.

Referring to FIG. 3, the electronic apparatus 100 may recognize a userby using face information. For example, the electronic apparatus 100 mayregister face information regarding a plurality of users. The electronicapparatus 100 may extract face information from each of images A1 to A5regarding the plurality of users by using a first recognition model or asecond recognition model. Face information extracted from the image A1of user A may be denoted by A2, face information extracted from theimage B1 of user B may be denoted by B2, face information extracted fromthe image C1 of user C may be denoted by C2, face information extractedfrom the image D1 of user D may be denoted by D2, and face informationextracted from the image E1 of user E may be denoted by E2. The faceinformation A2 to E2 may be represented as a feature matrix, a featurevector, a number, etc., but the present disclosure is not limitedthereto.

The electronic apparatus 100 may store the extracted face informationmatched with each user in a database 350.

The electronic apparatus 100 may obtain a user image 310 intended to berecognized. The electronic apparatus 100 may detect a face region 320from the user image 310. Extraction of a face region refers toextraction of location information regarding a user's face from the userimage 310, and may be performed by using various prior algorithms (e.g.,a Viola-Jones algorithm, a deep-learning-based algorithm, etc.). Theelectronic apparatus 100 may divide the user image 310 into a pluralityof pixel block units, may calculate a representative pixel value foreach pixel block, and may detect the face region 320 based on thecalculated representative pixel value and a location of each pixelblock. However, the present disclosure is not limited thereto.

When the face region 320 is detected, the electronic apparatus 100 mayextract face information from the face region 320. Extraction of faceinformation may be performed by using various prior algorithms (e.g., atemplate matching technique, a deep-learning-based algorithm, etc.). Forexample, the electronic apparatus 100 may extract feature parametersindicating facial features from the face region 320 and determine faceinformation based on the feature parameters. The electronic apparatus100 may extract various feature parameters such as a face shape or size,a face length, a face width, a distance between eyebrows, a nose bridgelength, a lip tail angle, a lip length, an eye size, an eye location, aneye tail angle, a nose size, an ear location, an eyebrow thickness, aneyebrow location, an eyebrow length, etc. However, the presentdisclosure is not limited thereto.

Referring to FIG. 3, the electronic apparatus 100 may extract faceinformation 330 from the face region 320 of the user image 310.

The electronic apparatus 100 may compare the face information 330extracted from the user image 310 with face information registered in adatabase and determine a user corresponding to the face information 330.For example, the electronic apparatus 100 may recognize a usercorresponding to the user image 310 as user C when it is determined thatthe face information 330 and the face information C2 registered in thedatabase 350 are in the same or similar range.

FIG. 4 is a diagram of a method of extracting additional featureinformation, according to an embodiment of the present disclosure.

Referring to FIG. 4, the electronic apparatus 100 may detect a personregion in a user image and may extract additional feature informationfrom the person region. In addition, additional feature information mayinclude appearance information and behavior information of a user. Theappearance information of the user may include a body shape, ahairstyle, a hair color, whether to wear glasses, a clothing style, anda side or front view of the user. In addition, the behavior informationof the user may include a gait and a behavior habit of the user.However, the present disclosure is not limited thereto.

Referring to FIG. 4, the electronic apparatus 100 may detect a personregion 401 in a first user image 410. For example, the electronicapparatus 100 may detect a face region 402, may detect a boundary lineof the person region 401 based on the detected face region 402, and maydetect the person region 401 according to the detected boundary line.Alternatively, a partial body region other than the face region 402 maybe detected, and the person region 401 may be detected based on thedetected partial body region. However, the present disclosure is notlimited thereto, and the person region 401 may be detected by usingvarious prior algorithms (for example, a human detection technique,etc.).

In addition, when the person region 401 is detected, the electronicapparatus 100 may divide the person region 401 into regions for bodyparts. For example, the electronic apparatus 100 may divide an entireregion of the user into partial regions such as a head region, an armregion, a leg region, a face region, and a trunk region of the user. Theelectronic apparatus 100 may extract additional feature information fromthe regions for body parts.

For example, information such as a hairstyle 411 (e.g., a hair length,whether hair is wavy or not, whether to wear bangs, etc.), a hair color,and a hair accessory may be extracted from the head region of the user.Alternatively, information such as whether to wear glasses, a facialhair shape, and a facial hair location may be extracted from the faceregion of the user. Alternatively, a clothing style 412, body shapeinformation, etc. may be extracted from the arm or trunk region of theuser.

The electronic apparatus 100 may also detect person regions 403 and 404in a second user image 420 including a side view of the user or a thirduser image 430 including a rear view of the user and may extract sideview information or rear view information of the user from the detectedperson regions 403 and 404.

The electronic apparatus 100 may also detect person regions 405 and 406in a fourth user image 440 or a fifth user image 450 including theuser's whole body and may extract information such as a body shape and abody proportion of the user from the detected person regions 405 and406. Information such as a gait and a behavior habit of the user may beextracted based on the images 440 and 450 regarding the user's wholebody which are obtained in real time.

Based on additional feature information extracted from the user image,the electronic apparatus 100 may allow a first recognition model or asecond recognition model to additionally learn and thus may update thefirst recognition model or the second recognition model.

In addition, the electronic apparatus 100 may match the additionalfeature information extracted from the user image with the user andstore the matched additional feature information in a database. When auser image is obtained, a user corresponding to additional featureinformation extracted from the obtained user image may be determined bycomparing the additional feature information extracted from the obtaineduser image with the additional feature information stored in thedatabase.

FIG. 5 is a diagram showing a recognition result of a first recognitionmodel and a recognition result of a second recognition model, accordingto an embodiment of the present disclosure.

Referring to FIG. 5, the first recognition model 30 may be a model ofextracting face information from a face region of a user image andrecognizing a user based on the extracted face information. The secondrecognition model 50 may be a model of extracting at least one of faceinformation and additional feature information from a person region ofthe user image and recognizing a user based on the extracted faceinformation and additional feature information. For example, the secondrecognition model 50 may be a model of recognizing a user by using onlyadditional feature information of the user without face information ofthe user.

The first recognition model 30 may be additionally learned based on faceinformation of a user and be updated. When the first recognition model30 is additionally learned based on face information of a user, anaccuracy of recognizing a user based on face information may increase.

In addition, the second recognition model 50 may be a model extended byallowing the first recognition model 30 to additionally learn based onadditional feature information. For example, the second recognitionmodel 50 may be, but is not limited to, a model with the firstrecognition model 30 having additionally learned based on additionalfeature information of a user. The second recognition model 50 may belearned based on additionally collected additional feature informationof a user and be updated.

Referring to FIG. 5, the electronic apparatus 100 may obtain a userimage 510. The user image 510 may be an image including a user A 521, auser B 522, and a user C 523. In the user image 510, a face of the userA 521 appears clear from the front, and only a rear view of the user B522 is shown while a face of the user B 522 is not shown. In addition,the user C 523 is far away, and thus, a face of the user C 523 appearsblurry.

When the user image 510 is input to the first recognition model 30, thefirst recognition model 30 recognizes only the user A 521 where a faceregion may be detected because the first recognition model 30 recognizesa user based on face information, and the user B 522 and the user C 523may not be recognized because face regions thereof may not be detected.

On the other hand, when the user image 510 is input to the secondrecognition model 50, all of the user A 521, the user B 522, and theuser C 523 may be recognized. For example, the electronic apparatus 100may recognize the user A 521 based on face information extracted from aface region of the user A 521 and hair style information, clothing styleinformation, etc. extracted from a person region of the user A 521. Theelectronic apparatus 100 may recognize the user B 522 based on rear viewinformation, clothing style information, etc. extracted from a personregion of the user B 522. The electronic apparatus 100 may recognize theuser C 523 based on body shape information, gait information, etc.extracted from a person region of the user C 523.

Accordingly, the electronic apparatus 100 may recognize a user byextracting additional feature information of the user even when a faceregion of the user is not detected in a user image (for example, in thecase of a side or rear view image of the user, an image with the userfar away, etc.).

FIG. 6 is a flowchart of a method in which an electronic apparatusupdates a second recognition model, according to an embodiment of thepresent disclosure.

Referring to FIG. 6, in operation S610, the electronic apparatus 100 mayobtain a user image.

In operation S620, the electronic apparatus 100 may extract additionalfeature information from a person region of the obtained user image. Amethod in which an electronic apparatus extracts additional featureinformation of a user has been described above with reference to FIG. 4.

In operation S630, the electronic apparatus 100 may recognize a userbased on the additional feature information. For example, the electronicapparatus 100 may determine a user corresponding to the extractedadditional feature information by comparing the extracted additionalfeature information with additional feature information matching each ofa plurality of users stored in a database. However, the presentdisclosure is not limited thereto.

In operation S640, the electronic apparatus 100 may evaluate arecognition model based on a recognition result. For example, theelectronic apparatus 100 may determine a recognition error when the userrecognized based on the additional feature information in operation S630is compared with a user recognized based on face feature information andthe users are determined to be different from each other. Alternatively,the electronic apparatus 100 may determine a recognition error when theuser recognized based on the additional feature information and a userrecognized based on face feature information are the same as each otherbut, as a result of recognition based on the additional featureinformation, a probability of the recognized user is less than apredetermined value.

For example, the electronic apparatus 100 may finally recognize user Aas a probability of the user A is determined as 62% and a probability ofthe user B is determined as 37% as a result of recognizing a user basedon additional feature information, and may finally recognize user A as aprobability of the user A is determined as 95% as a result ofrecognition based on face feature information. Although recognitionresults are the same, as a result of recognition based on additionalfeature information, a probability of user A is less than apredetermined value, 80%, and thus, the electronic apparatus 100 maydetermine a recognition error.

However, a method of evaluating a second recognition model is notlimited thereto, and the second recognition model may be evaluated invarious ways.

In operation S650, the electronic apparatus 100 may determine, based onan evaluation result of the second recognition model, whether it isnecessary to update the second recognition model. For example, theelectronic apparatus 100 may count the number of recognition errorswhenever a user is recognized by using the second recognition model, andthus, when the number of recognition errors is equal to or greater thana predetermined number, may determine that it is necessary to update thesecond recognition model. Alternatively, the electronic apparatus 100may calculate a recognition error rate, and thus, when the recognitionerror rate is equal to or greater than a predetermined value, maydetermine that it is necessary to update the second recognition model.However, the present disclosure is not limited thereto.

In operation S660, when the second recognition model is necessary toupdate, the electronic apparatus 100 may allow the second recognitionmodel to additionally learn by using collected additional featureinformation and thus may update the second recognition model.

The electronic apparatus 100 may update the second recognition model inreal time. Alternatively, the electronic apparatus 100 may storecollected additional feature information and periodically update therecognition model by using the stored additional feature information, ormay update the second recognition model when the electronic apparatus100 is in an idle state (for example, when the electronic apparatus 100is being charged), when a user request is input, or when it is apredetermined time. However, the present disclosure is not limitedthereto.

In addition, the electronic apparatus 100 may allow the secondrecognition model to entirely learn or may allow only a portion of thesecond recognition model to learn. However, the present disclosure isnot limited thereto.

FIG. 7 is a flowchart of a method in which an electronic apparatusupdates a second recognition model, according to an embodiment of thepresent disclosure.

Referring to FIG. 7, in operation S710, the electronic apparatus 100 mayobtain user images regarding the same user in real time.

In operation S720, the electronic apparatus 100 may determine whether itis possible to detect a face region in an image obtained in real time,and in operation S730, when it is not possible to detect a face region,the electronic apparatus 100 may detect a person region in the userimage and may extract additional feature information of the user fromthe person region. A method in which an electronic apparatus extractsadditional feature information of a user has been described above withreference to FIG. 4.

In operation S740, the electronic apparatus 100 may recognize, based onthe extracted additional feature information, a first user in the userimage.

In operation S750, when it is possible to detect a face region, theelectronic apparatus 100 may extract face information from the faceregion, and in operation S760, the electronic apparatus 100 mayrecognize a second user based on the face information. A method in whichan electronic apparatus recognizes a user based on face information hasbeen described above with reference to FIG. 3.

In operation S770, the electronic apparatus 100 may determine whetherthe recognized first user and second user are the same as each other,and when the first user and the second user are not the same as eachother, the electronic apparatus 100 may determine that a secondrecognition model is necessary to update.

In operation S780, the electronic apparatus 100 may update the secondrecognition model by using the additional feature information. Forexample, the electronic apparatus 100 may allow the second recognitionmodel to learn by matching the additional feature information extractedin operation S730 with the second user recognized in operation S760 andthus may update the second recognition model. However, the presentdisclosure is not limited thereto.

In addition, the electronic apparatus 100 may update the secondrecognition model in real time, or may store additional featureinformation and then may update the second recognition model at apredetermined time by using the stored additional feature information.The electronic apparatus 100 may allow the second recognition model toentirely learn or may allow only a portion of the second recognitionmodel to learn. However, the present disclosure is not limited thereto.

FIG. 8 is a diagram of a method in which an electronic apparatusdetermines whether to update a recognition model, according to anembodiment of the present disclosure.

Referring to FIG. 8, the electronic apparatus 100 may obtain a userimage 810. The user image 810 may be an image captured in the electronicapparatus 100 or an image received from an external apparatus. Inaddition, the user image 810 may be an image where a face region may bedetected.

The electronic apparatus 100 may detect a face region 802 in the userimage 810 and may extract face information 820 from the detected faceregion 802. This has been described above with reference to FIG. 3.

Referring to FIG. 8, the electronic apparatus 100 may recognize user Abased on the face information 820.

In addition, the electronic apparatus 100 may detect a person region 801in the user image 810 and may extract additional feature information 830from the detected person region 801. This has been described withreference to FIG. 4, and thus, a detailed description thereof isomitted.

The electronic apparatus 100 may recognize “user B” based on theextracted additional feature information 830.

Referring to FIG. 8, when a user recognized in the user image 810 basedon the face information 820 is user A, and a user recognized in the userimage 810 based on the additional feature information 830 is user B, theelectronic apparatus 100 may determine that a recognition result of thesecond recognition model 50 is not accurate. In addition, the electronicapparatus 100 may determine that the recognition model is necessary toupdate, and may allow the second recognition model 50 to additionallylearn by matching the extracted additional feature information 830 withuser A and thus may update the second recognition model 50. However, thepresent disclosure is not limited thereto.

FIG. 9 is a diagram of a method in which an electronic apparatusdetermines whether to update a recognition model, according to anembodiment of the present disclosure.

Referring to FIG. 9, the electronic apparatus 100 may obtain user imagesregarding the same user in real time. For example, referring to FIG. 9,the electronic apparatus 100 may obtain a first user image 910 and asecond user image 920 regarding the same user.

The first user image 910 may be an image where a face region is notdetected. The electronic apparatus 100 may detect a person region 901 inthe first user image 910 and may extract additional feature information915 of the user from the person region 901. For example, the electronicapparatus 100 may extract rear view information of the user, hair styleinformation of the user, hair color information, clothing styleinformation, clothing color information, body shape or body proportioninformation of the user, etc. from the person region 901. The electronicapparatus 100 may recognize user A in the first user image 910 based onthe extracted additional feature information 915.

The second user image 920 may be an image where a face region isdetected. The electronic apparatus 100 may detect a face region 902 inthe second user image 920 and may extract face information 925 of theuser from the face region 902. For example, the electronic apparatus 100may extract various feature parameters such as a face shape or size, aface length, a face width, a distance between eyebrows, a nose bridgelength, a lip tail angle, a lip length, an eye size, an eye location, aneye tail angle, a nose size, an ear location, an eyebrow thickness, aneyebrow location, an eyebrow length, etc. from the face region 902 andmay determine face information based on the extracted featureparameters.

The electronic apparatus 100 may recognize user B in the second userimage 920 based on extracted face information of the user. However, thepresent disclosure is not limited thereto, and the electronic apparatus100 may detect a person region 903 in the second user image 920, extractadditional feature information of the user from the person region 903,and recognize the user based on the face information 925 and theadditional feature information.

When user A recognized in the first user image 910 and user B recognizedin the second user image 920 are not the same as each other, theelectronic apparatus 100 may allow the second recognition model 50 toadditionally learn by using the additional feature information 915extracted from the first user image 910 and thus may update the secondrecognition model 50. In this regard, the electronic apparatus 100 mayallow the second recognition model 50 to additionally learn by matchingthe additional feature information 915 with user B and thus may updatethe second recognition model 50. However, the present disclosure is notlimited thereto.

The electronic apparatus 100 may update the second recognition model 50in real time. Alternatively, the electronic apparatus 100 may store theadditional feature information 915 and periodically update therecognition model by using the stored additional feature information915, or may update the second recognition model 50 when the electronicapparatus 100 is in an idle state (e.g., when the electronic apparatus100 is being charged), when a user request is input, or when it is apredetermined time. However, the present disclosure is not limitedthereto.

In addition, the electronic apparatus 100 may allow the secondrecognition model 50 to entirely learn or may allow only a portion ofthe second recognition model 50 to learn. However, the presentdisclosure is not limited thereto.

FIG. 10 is a block diagram of a structure of an electronic apparatus,according to an embodiment of the present disclosure.

Referring to FIG. 10, the electronic apparatus 100 may include a camera140, a processor 120, and a memory 130.

The camera 140 may obtain an image frame such as a still image or videothrough an image sensor. An image captured through the image sensor maybe processed through the processor 120. The camera 140 according to anembodiment may obtain a user image by capturing an image of a userintended to be recognized.

The processor 120 may execute one or more programs stored in the memory130. The processor 120 may include a single core, dual core, triplecore, quad core, and multiple cores. In addition, the processor 120 mayinclude a plurality of processors. For example, the processor 120 may beimplemented as a main processor (not shown) and a sub processor (notshown) operating in a sleep mode.

The memory 130 may store various data, programs, or applications fordriving and controlling the electronic apparatus 100.

The program stored in the memory 130 may include one or moreinstructions. The program (one or more instructions) or the applicationstored in the memory 130 may be executed by the processor 120.

The processor 120 may extract face information from a face region of auser image by using a first recognition model stored in the memory 130and may recognize a user by comparing the extracted face informationwith face information regarding a plurality of users registered in theelectronic apparatus 100. In addition, the processor 120 may detect aperson region in the user image by executing one or more instructionsstored in the memory 130 and may extract additional feature informationsuch as appearance information or behavior information of the user fromthe person region. The processor 120 may match the extracted additionalfeature information with the user by executing one or more instructionsstored in the memory 130 and thus may allow the first recognition modelto additionally learn. The processor 120 may store an additionallylearned second recognition model in the memory 130. The processor 120may use the additionally learned second recognition model by executingone or more instructions stored in the memory 130 and thus may recognizethe user from the person region of the user image.

The processor 120 may evaluate a recognition result of the secondrecognition model by executing one or more instructions stored in thememory 130 and thus may determine whether the second recognition modelis necessary to update, and when it is determined that the secondrecognition model is necessary to update, the processor 120 may allowthe second recognition model to additionally learn based on additionalfeature information collected in real time and thus may update thesecond recognition model.

FIG. 11 is a block diagram of a processor according to an embodiment ofthe present disclosure.

Referring to FIG. 11, the processor 120 may include a data learning unit1300 and a data recognition unit 1400.

The data learning unit 1300 may generate a data recognition model orallow a data recognition model to learn so that the data recognitionmodel may have a criterion for recognizing a user in a user image. Thedata learning unit 1300 may generate a data recognition model having acriterion of determination by applying learning data to the datarecognition model in order to recognize a user in a user image.

The data learning unit 1300 may generate a data recognition model orallow a data recognition model to learn by using learning data relatedto an image. The data recognition model may include a first recognitionmodel and a second recognition model.

The data recognition unit 1400 may recognize a user based on recognitiondata. The data recognition unit 1400 may recognize a user from apredetermined user image by using a learned data recognition model. Thedata recognition unit 1400 may obtain predetermined data (e.g., a userimage) according to a predetermined criterion by learning and mayrecognize a user based on the user image by using a data recognitionmodel with the obtained data as an input value. For example, a user maybe recognized based on face information of the user extracted from aface region of the user image, or may be recognized based on additionalfeature information such as appearance information or behaviorinformation of the user extracted from a person region of the userimage. In addition, a result value output by the data recognition modelwith the obtained data as an input value may be used to update the datarecognition model.

At least a portion of the data learning unit 1300 and at least a portionof the data recognition unit 1400 may be implemented as a softwaremodule or manufactured in the form of at least one hardware chip and beequipped in an electronic apparatus. For example, at least one of thedata learning unit 1300 and the data recognition unit 1400 may bemanufactured in the form of an exclusive hardware chip for artificialintelligence (AI) or may be manufactured as a portion of an existinggeneral-use processor (e.g., a central processing unit (CPU) or anapplication processor) or a graphic exclusive processor (e.g., agraphics processing unit (GPU)) and be equipped in various kinds ofelectronic apparatuses described above.

The exclusive hardware chip for AI, which is an exclusive processorspecified for probability computation, has a higher parallel processingperformance than the existing general-use processor and thus may quicklyprocess a computation job of an AI field such as machine learning. Whenthe data learning unit 1300 and the data recognition unit 1400 areimplemented as a software module (or a program module including aninstruction), the software module may be stored in a non-transitorycomputer-readable recording medium. In this case, the software modulemay be provided by an operating system (OS) or may be provided by apredetermined application. Alternatively one portion of the softwaremodule may be provided by the OS, and the other portion thereof may beprovided by the predetermined application.

The data learning unit 1300 and the data recognition unit 1400 may beinstalled in one electronic apparatus or may be respectively installedin individual electronic apparatuses. For example, one of the datalearning unit 1300 and the data recognition unit 1400 may be included inan electronic apparatus, and the other may be included in a server. Inaddition, the data learning unit 1300 and the data recognition unit 1400may be connected in a wired or wireless manner to provide modelinformation constructed by the data learning unit 1300 to the datarecognition unit 1400 and provide data input to the data recognitionunit 1400 to the data learning unit 1300 as additional learning data.

FIG. 12 is a block diagram of a data learning unit according to anembodiment of the present disclosure.

Referring to FIG. 12, the data learning unit 1300 may include a dataobtaining unit 1310 and a model learning unit 1340. In addition, thedata learning unit 1300 may selectively further include at least one ofa preprocessor 1320, a learning data selection unit 1330, and/or a modelevaluation unit 1350. The data obtaining unit 1310 may obtain learningdata required for learning for recognizing a user.

Data collected or tested by the data learning unit 1300 or amanufacturer of an electronic apparatus may be used as the learningdata. Alternatively, the learning data may include image data generatedfrom a user image input through a camera according to the presentdisclosure. In this regard, although the camera may be included in theelectronic apparatus, this is merely an embodiment, and image dataobtained through an external camera may be used as the learning data.

The data obtaining unit 1310 may obtain a plurality of user images. Forexample, the data obtaining unit 1310 may receive a user image through acamera of an electronic apparatus including the data learning unit 1300.Alternatively, a user image may be received through an externalapparatus capable of communicating with an electronic apparatusincluding the data learning unit 1300.

The model learning unit 1340 may learn a criterion of how a datarecognition model will recognize a user in a user image by usinglearning data. For example, the model learning unit 1340 may allow adata recognition model to learn through supervised learning in which atleast a portion of learning data is used as a criterion ofdetermination. Alternatively, the model learning unit 1340 may allow adata recognition model to learn through unsupervised learning in which acriterion of detecting a face region or a person region in a user image,a criterion of extracting face information from the face region, and acriterion of extracting additional feature information from the personregion are discovered, for example, by learning by itself using learningdata without supervision.

In addition, the model learning unit 1340 may learn a criterion of whichlearning data will be used to recognize a user.

The model learning unit 1340 according to an embodiment of the presentdisclosure may generate a data recognition model or allow a datarecognition model to learn by using learning data related to a criterionof detecting a face region or a person region in a user image, acriterion of extracting face information from the face region, and acriterion of extracting additional feature information from the personregion. In this case, when the data recognition model is allowed tolearn through supervised learning, as criteria of determination, thecriterion of detecting a face region or a person region in a user image,the criterion of extracting face information from the face region, andthe criterion of extracting additional feature information from theperson region may be added as learning data.

For example, the model learning unit 1340 may generate a datarecognition model or allow a data recognition model to learn by usinglearning data related to a criterion of detecting a face region or aperson region in a user image, a criterion of extracting faceinformation from the face region, and a criterion of extractingadditional feature information from the person region. In addition, themodel learning unit 1340 may learn so as to recognize a user based onface information of a plurality of users stored in a database and faceinformation of a user extracted from a user image. Alternatively, themodel learning unit 1340 may learn so as to recognize a user based onadditional feature information of a plurality of users stored in adatabase and additional feature information of a user extracted from auser image.

The model learning unit 1340 may allow a data recognition model thatrecognizes a user in a user image to learn by using learning data. Inthis case, the data recognition model may be a previously constructedmodel. For example, the data recognition model may be a model previouslyconstructed by receiving basic learning data (e.g., a sample image,etc.).

The data recognition model may be constructed by taking into account anapplication field of the data recognition model, a purpose of learning,or computer performance of an apparatus. The data recognition model maybe, for example, a model based on neural network. For example, a modelsuch as DNN, RNN, or BRDNN may be used as the data recognition model,but the present disclosure is not limited thereto.

When there are a plurality of previously constructed data recognitionmodels, the data learning unit 1340 may determine a data recognitionmodel in which input learning data and basic learning data have a highrelevance as a data recognition model which will learn. In this case,the basic learning data may be previously classified according to typesof data, and the data recognition model may be previously constructedaccording to types of data. For example, the basic learning data may bepreviously classified according to various criteria such as an areawhere learning data is generated, a time when learning data isgenerated, a size of learning data, a genre of learning data, agenerator of learning data, and a type of an object within learningdata.

The model learning unit 1340 may allow a data recognition model to learnby using a learning algorithm, etc. including, for example, an errorback-propagation method or a gradient descent method.

The model learning unit 1340 may allow a data classification model tolearn, for example, through supervised learning with learning data as aninput value. The model learning unit 1340 may allow a dataclassification model to learn, for example, through unsupervisedlearning in which a criterion for judging a situation is discovered bylearning by itself a type of data required of judging a situationwithout supervision. The model learning unit 1340 may allow a datarecognition model to learn, for example, through reinforcement learningusing a feedback regarding whether a result of image classificationaccording to learning is correct.

When a data recognition model is learned, the model learning unit 1340may store the learned data recognition model. In this case, the modellearning unit 1340 may store the learned data recognition model in amemory of an electronic apparatus including the data recognition unit1400. The model learning unit 1340 may store a learned dataclassification model in a memory of an electronic apparatus includingthe data recognition unit 1400 described below. The model learning unit1340 may store a learned data classification model in a memory of aserver connected to an electronic apparatus via a wired or wirelessnetwork.

A memory in which the learned data recognition model is stored may alsostore, for example, a command or data related to at least one othercomponent of an electronic apparatus. In addition, the memory may storesoftware and/or a program. The program may include, for example, akernel, middleware, an application programming interface (API) and/or anapplication program (or “application”).

The data learning unit 1300 may further include the preprocessor 1320and the learning data selection unit 1330 to improve a recognitionresult of a data recognition model or save a resource or time requiredto generate a data recognition model.

The preprocessor 1320 may preprocess data obtained in the data obtainingunit 1310 to use the data in learning for recognizing a user. Forexample, the preprocessor 1320 may process the obtained data into apredefined format to facilitate use of data for learning of a datarecognition model. The preprocessed data may be provided to the modellearning unit 1340 as learning data.

The learning data selection unit 1330 may selectively select learningdata required for learning from among the preprocessed data. Theselected learning data may be provided to the model learning unit 1340.The learning data selection unit 1330 may select learning data requiredfor learning from among the preprocessed data according to a presetcriterion of selection. In addition, the learning data selection unit1330 may select learning data required for learning according to acriterion of selection preset by learning in the model learning unit1340.

The data learning unit 1300 may further include the model evaluationunit 1350 to improve a recognition result of a data recognition model.The model evaluation unit 1350 may input evaluation data to a datarecognition model, and if a recognition result output from theevaluation data does not satisfy a certain criterion, the modelevaluation unit 1350 may allow the model learning unit 1340 to learnagain. In this case, the evaluation data may be preset data forevaluating the data recognition model.

For example, when the number or percentage of evaluation data of which arecognition result is not accurate among classification results of thelearned data recognition model for evaluation data exceeds a presetthreshold, the model evaluation unit 1350 may evaluate that a certaincriterion is not satisfied. For example, when the certain criterion isdefined as 2%, if the learned data recognition model outputs wrongrecognition results for more than 20 evaluation data among a total of1000 evaluation data, the model evaluation unit 1350 may evaluate thatthe learned data recognition model is not suitable.

When there are a plurality of learned data recognition models, the modelevaluation unit 1350 may evaluate whether each of the learned datarecognition models satisfies a certain criterion and may determine amodel satisfying the certain criterion as a final data recognitionmodel. In this case, when a plurality of models satisfy the certaincriterion, the model evaluation unit 1350 may determine any one model ora predetermined number of models preset in an order of higher evaluationscore as the final data recognition model.

At least one of the data obtaining unit 1310, the preprocessor 1320, thelearning data selection unit 1330, the model learning unit 1340, and themodel evaluation unit 1350 in the data learning unit 1300 may bemanufactured in the form of at least one hardware chip and be equippedin an electronic apparatus. For example, at least one of the dataobtaining unit 1310, the preprocessor 1320, the learning data selectionunit 1330, the model learning unit 1340, and the model evaluation unit1350 may be manufactured in the form of an exclusive hardware chip forAI or may be manufactured as a portion of an existing general-useprocessor (e.g., a CPU or an application processor) or a graphicexclusive processor (e.g., a GPU) and be equipped in various types ofelectronic apparatuses described above.

In addition, the data obtaining unit 1310, the preprocessor 1320, thelearning data selection unit 1330, the model learning unit 1340, and themodel evaluation unit 1350 may be installed in one electronic apparatusor may be respectively installed in individual electronic apparatuses.For example, some of the data obtaining unit 1310, the preprocessor1320, the learning data selection unit 1330, the model learning unit1340, and the model evaluation unit 1350 may be included in anelectronic apparatus, and the other some may be included in a server.

In addition, at least one of the data obtaining unit 1310, thepreprocessor 1320, the learning data selection unit 1330, the modellearning unit 1340, and the model evaluation unit 1350 may beimplemented as a software module. When at least one of the dataobtaining unit 1310, the preprocessor 1320, the learning data selectionunit 1330, the model learning unit 1340, and the model evaluation unit1350 is implemented as a software module (or a program module includingan instruction), the software module may be stored in a non-transitorycomputer-readable recording medium. In addition, in this case, at leastone software module may be provided by an OS or may be provided by apredetermined application. Alternatively, one portion of at least onesoftware module may be provided by the OS, and the other portion may beprovided by the predetermined application.

FIG. 13 is a block diagram of a data recognition unit according to anembodiment of the present disclosure.

Referring to FIG. 13, the data recognition unit 1400 may include a dataobtaining unit 1410 and a recognition result providing unit 1440. Thedata recognition unit 1400 may selectively further include at least oneof a preprocessor 1420, a recognition data selection unit 1430, and/or amodel updating unit 1450.

The data obtaining unit 1410 may obtain data required for userrecognition, and the preprocessor 1420 may preprocess the obtained datato use the data obtained for user recognition. The preprocessor 1420 mayprocess the obtained data into a preset format such that the recognitionresult providing unit 1440 described below uses the data obtained foruser recognition.

The recognition result providing unit 1440 may recognize a user byapplying selected data to a data recognition model. The recognitionresult providing unit 1440 may provide a recognition result according toa purpose of data recognition. The recognition result providing unit1440 may use data selected by the recognition data selection unit 1430as an input value and thus may apply the selected data to the datarecognition model. In addition, the recognition result may be determinedby the data recognition model.

For example, the recognition result providing unit 1440 may displayinformation regarding a recognized user, or may output an alarm or awarning message when the recognized user is not a fair user.Alternatively, when the recognized user is a fair user, a predeterminedservice may be provided.

The data recognition unit 1400 may further include the preprocessor 1420and the recognition data selection unit 1430 to improve a recognitionresult of a data recognition model or save a resource or time forproviding a recognition result.

The preprocessor 1420 may preprocess data obtained in the data obtainingunit 1410 to learn a criterion of determination for recognizing a userin a user image. The preprocessor 1420 may process the obtained datainto a predefined format to facilitate use of data for learning acriterion of determination for user recognition.

The recognition data selection unit 1430 may select data required foruser recognition from among the preprocessed data. The selected data maybe provided to the recognition result providing unit 1440. Therecognition data selection unit 1430 may select a portion or all of thepreprocessed data according to a preset criterion for user recognition.In addition, the recognition data selection unit 1430 may select dataaccording to a criterion preset by learning in the model learning unit1340.

The model updating unit 1450 may update a data recognition model basedon evaluation regarding a recognition result provided by the recognitionresult providing unit 1440. For example, the model updating unit 1450may allow the model learning unit 1340 to update the data recognitionmodel by providing the recognition result provided by the recognitionresult providing unit 1440 to the model learning unit 1340.

At least one of the data obtaining unit 1410, the preprocessor 1420, therecognition data selection unit 1430, the recognition result providingunit 1440, and/or the model updating unit 1450 in the data recognitionunit 1400 may be implemented as a software module or may be manufacturedin the form of at least one hardware chip and be equipped in anelectronic apparatus. For example, at least one of the data obtainingunit 1410, the preprocessor 1420, the recognition data selection unit1430, the recognition result providing unit 1440, and/or the modelupdating unit 1450 may be manufactured in the form of an exclusivehardware chip for AI or may be manufactured as a portion of an existinggeneral-use processor (e.g., a CPU or an application processor) or agraphic exclusive processor (e.g., a GPU) and be installed in varioustypes of electronic apparatuses described above.

In addition, the data obtaining unit 1410, the preprocessor 1420, therecognition data selection unit 1430, the recognition result providingunit 1440, and the model updating unit 1450 may be installed in oneelectronic apparatus or may be respectively installed in individualelectronic apparatuses. For example, some of the data obtaining unit1410, the preprocessor 1420, the recognition data selection unit 1430,the recognition result providing unit 1440, and the model updating unit1450 may be included in an electronic apparatus, and the remainder maybe included in a server.

In addition, at least one of the data obtaining unit 1410, thepreprocessor 1420, the recognition data selection unit 1430, therecognition result providing unit 1440, and the model updating unit 1450may be implemented as a software module. When at least one of the dataobtaining unit 1410, the preprocessor 1420, the recognition dataselection unit 1430, the recognition result providing unit 1440, and/orthe model updating unit 1450 is implemented as a software module (or aprogram module including an instruction), the software module may bestored in a non-transitory computer-readable recording medium. Inaddition, in this case, at least one software module may be provided byan OS or may be provided by a predetermined application. Alternatively,one portion of at least one software module may be provided by the OS,and the other portion may be provided by the predetermined application.

FIG. 14 is a diagram of an example in which data is learned andrecognized by an electronic apparatus and a server interworking witheach other, according to an embodiment of the present disclosure.

Referring to FIG. 14, a server 2000 may learn a criterion forrecognizing a user by analyzing a user image, and the electronicapparatus 100 may recognize a user based on a learning result by a datalearning unit 2300 of the server 2000. The data learning unit 2300 mayinclude a data obtaining unit 2310, a preprocessor 2320, a learning dataselection unit 2330, a model learning unit 2340, and/or a modelevaluation unit 2350.

In this case, the model learning unit 2340 of the server 2000 mayperform functions of the model learning unit 1340 shown in FIG. 12. Themodel learning unit 2340 of the server 2000 may analyze an image andthus may learn a criterion regarding which data will be used torecognize a user and how the user will be recognized by using data. Themodel learning unit 2340 may obtain data which will be used in learningand may learn a criterion for user recognition by applying the obtaineddata to a data recognition model described below.

The recognition result providing unit 1440 of the electronic apparatus100 may judge a situation by applying data selected by the recognitiondata selection unit 1430 to the data recognition model generated by theserver 2000. For example, the recognition result providing unit 1440 maytransmit data selected by the recognition data selection unit 1430 tothe server 2000, and the server 2000 may request recognizing a user byapplying data selected by the recognition data selection unit 1430 tothe recognition model. In addition, the recognition result providingunit 1440 may receive information regarding a user recognized by theserver 2000 from the server 2000.

Alternatively, the recognition result providing unit 1440 of theelectronic apparatus 100 may receive the recognition model generated bythe server 2000 from the server 2000, may analyze an image by using thereceived recognition model, and may recognize a user. In this case, therecognition result providing unit 1440 of the electronic apparatus 100may recognize the user by applying data selected by the recognition dataselection unit 1430 to the data recognition model received from theserver 2000.

FIG. 15 is a block diagram of a structure of an electronic apparatus300, according to an embodiment of the present disclosure. Theelectronic apparatus 300 of FIG. 15 may be an embodiment of theelectronic apparatus 100 of FIG. 1.

Referring to FIG. 15, the electronic apparatus 300 may include aprocessor 330, a sensor 320, a communicator 340, an output interface350, a user input interface 360, an audio/video (A/V) input interface370, and a storage unit 380.

The processor 330, the storage unit 380, and a camera 371 of FIG. 15 mayrespectively correspond to the processor 120, the memory 130, and thecamera 140 of FIG. 10. A description of those components shown in FIG.15 that have already been described with reference to FIG. 10 will beomitted.

The communicator 340 may include one or more components forcommunication between the electronic apparatus 300 and an externalapparatus (e.g., a server). For example, the communicator 340 mayinclude a short-range wireless communicator 341, a mobile communicator342, and a broadcast receiver 343.

The short-range wireless communicator 341 may include a Bluetoothcommunicator, a near field communicator, a wireless local area network(WLAN) communicator, a Zigbee communicator, an infrared data association(IrDA) communicator, a WFD (Wi-Fi Direct) communicator, an ultrawideband (UWB) communicator, and an Ant+ communicator, but is notlimited thereto.

The mobile communicator 342 may transmit and receive a wireless signalto and from at least one of a base station, an external terminal, and aserver on a mobile communication network. In this regard, the wirelesssignal may include a voice call signal, a video call signal, or varioustypes of data generated during text/multimedia messagetransmission/reception.

The broadcast receiver 343 may externally receive a broadcast signaland/or broadcast-related information through a broadcast channel. Thebroadcast channel may include a satellite channel and a terrestrialchannel. In some embodiments, the electronic apparatus 300 may omit thebroadcast receiver 343.

The communicator 340 may receive at least one user image from anexternal apparatus.

The output interface 350 may be used to output an audio signal, a videosignal, or a vibration signal and may include a display 351, a soundoutput interface 352, and a vibration motor 353.

The display 351 may generate a driving signal by converting an imagesignal, a data signal, an on-screen display (OSD) signal, a controlsignal, etc. processed in the processor 120. The display 351 may beimplemented as a plasma display panel (PDP), a liquid crystal display(LCD), an organic light-emitting diode (OLED), a flexible display, etc.and may also be implemented as a three-dimensional (3D) display. Inaddition, the display 351 may be configured as a touchscreen and be usedas an input apparatus in addition to an output apparatus.

The display 351 may display a user image. The image displayed on thedisplay 351 may be, but is not limited to, at least one of an imagecaptured in the electronic apparatus 300, an image stored in theelectronic apparatus 300, and an image received from an externalapparatus. In addition, the display 351 may display a user recognitionresult. For example, the display 351 may display information regarding arecognized user or may display a warning message when the recognizeduser is not a fair user. Alternatively, when the recognized user is nota registered user, the display 351 may display a recognition errormessage or may display a message asking whether to register a new user.However, the present disclosure is not limited thereto.

The sound output interface 352 may output audio data received from thecommunicator 340 or stored in the storage unit 380. In addition, thesound output interface 352 may output a sound signal related to afunction performed in the electronic apparatus 300 (e.g., a call signalreception sound, a message reception sound, or a notification sound).The sound output interface 352 may include a speaker, a buzzer, etc. Forexample, the sound output interface 352 may output an alarm when arecognized user is not an authorized user.

The vibration motor 353 may output a vibration signal. For example, thevibration motor 353 may output a vibration signal corresponding to anoutput of audio data or video data (e.g., a call signal reception sound,a message reception sound, etc.). In addition, the vibration motor 353may output a vibration signal when a touch is input to a touchscreen.

The processor 330 may control overall operations of the electronicapparatus 300. For example, the processor 330 may control thecommunicator 340, the output interface 350, the user input interface360, the sensor 320, the A/V input interface 370, and the like byexecuting programs stored in the storage unit 380.

The user input interface 360 refers to a means for inputting data forcontrolling the electronic device 300. The user input interface 360 mayinclude a key pad, a dome switch, a touch pad (e.g., a touch-typecapacitive touch pad, a pressure-type resistive overlay touch pad, aninfrared ray sensing touch pad, a surface acoustic wave conduction touchpad, an integration-type tension measurement touch pad, a piezoeffect-type touch pad, etc.), a jog wheel, and a jog switch, but is notlimited thereto.

The sensor 320 may include not only a sensor for sensing bodyinformation of a user but also a sensor for sensing a state of theelectronic apparatus 300 or a state of the vicinity of the electronicapparatus 300 and may transmit information sensed in the sensor to theprocessor 330.

The sensor 320 may include, but is not limited to, at least one of ageomagnetic sensor 321, an acceleration sensor 322, atemperature/humidity sensor 323, an infrared sensor 324, a gyroscopesensor 325, a position sensor 326 (e.g., a global positioning system(GPS)), a barometric pressure sensor 327, a proximity sensor 328, and ared, blue and green (RGB) sensor (illuminance sensor) 329. A function ofeach sensor may be intuitively inferred from the name by those ofordinary skill in the art, and thus, a detailed description thereof isomitted.

The A/V input interface 370 may be used to input an audio signal or avideo signal, and may include the camera 371 and a microphone 372. Thecamera 371 may obtain a video frame, such as a still image or a movingpicture, through an image sensor in a video call mode or a shootingmode. An image captured through the image sensor may be processedthrough the processor 330 or a separate image processor (not shown).

A video frame processed in the camera 371 may be stored in the storageunit 380 or may be transmitted to the outside through the communicator340. Two or more cameras 371 may be provided depending on configurationof the electronic apparatus 300.

The microphone 372 may receive an external acoustic signal and processthe external acoustic signal to electrical voice data. For example, themicrophone 372 may receive an acoustic signal from an external apparatusor a speaker. The microphone 372 may use various noise removalalgorithms to remove noise generated while receiving the externalacoustic signal.

The storage unit 380 may store programs for processing and controllingof the processor 330 and may store input/output data (e.g., anapplication, content, time slot information of an external device, anaddress book, etc.).

The storage unit 380 may include at least one type of storage mediumfrom among a flash memory type memory, a hard disk type memory, amultimedia card micro type memory, a card type memory (e.g., a securedigital (SD) or extreme digital (XD) memory, etc.), random access memory(RAM), static RAM (SRAM), read-only memory (ROM), electrically erasableprogrammable ROM (EEPROM), programmable ROM (PROM), a magnetic memory, amagnetic disc, and an optical disc. In addition, the electronicapparatus 300 may run a web storage that performs a storing function ofthe storage unit 380 on the Internet or a cloud server.

The programs stored in the storage unit 380 may be classified into aplurality of modules according to functions thereof, e.g., a userinterface (UI) module 381, a touchscreen module 382, and a notificationmodule 383.

The UI module 381 may provide a specialized UI or graphical UI (GUI)interworking with the electronic apparatus 300 for each application. Thetouchscreen module 382 may sense a touch gesture of a user on atouchscreen and may transmit information regarding the touch gesture tothe processor 330.

The touchscreen module 382 may recognize and analyze a touch code. Thetouchscreen module 382 may be configured as separate hardware includinga controller.

The notification module 383 may generate a signal for providingnotification of the occurrence of an event in the electronic apparatus300. Examples of the event occurring in the electronic apparatus 300 mayinclude call signal reception, message reception, key signal input, andschedule notification. The notification module 383 may output anotification signal in the form of a video signal through the display351, may output a notification signal in the form of an audio signalthrough the sound output interface 352, and may output a notificationsignal in the form of a vibration signal through the vibration motor353.

Each of the block diagrams of the electronic apparatuses 100 and 300shown in FIGS. 10 and 14 is a block diagram for an embodiment of thepresent invention. Components shown in the block diagrams may beintegrated, added, or omitted according to actually implementedspecification of the electronic apparatuses 100 and 300. Two or morecomponents may be combined into one component as necessary, or onecomponent may be divided into two or more components. In addition, afunction that is performed in each block is for describing embodiments,and a detailed operation or apparatus thereof does not limit the scopeof claims.

At least a portion of an apparatus (e.g., modules or functions thereof)or a method (e.g., operations) may be implemented as a command stored ina non-transitory computer-readable recording medium in the form of aprogram module. When the command is executed by a processor (e.g., theprocessor 330), the processor may perform a function corresponding tothe command.

The program described herein may be stored in a non-transitorycomputer-readable recording medium and be read and executed by acomputer to implement one or more embodiments of the present disclosure.

The non-transitory readable recording medium described herein not onlyrefers to a medium configured to semi-permanently store data andreadable by a device but also includes a register, a cache, a buffer,etc., and does not include a medium of transmission such as a signal orcurrent.

The programs described above may be stored in the non-transitoryreadable recording medium such as compact disc (CD), digital versatiledisc (DVD), a hard disk, a Blu-ray disk, universal serial bus (USB), anembedded memory (e.g., the memory 130), a memory card, ROM, or RAM andbe provided.

In addition, the methods according to the disclosed embodiments may beprovided as a computer program product. The computer program product mayinclude a software (S/W) program, a computer-readable storage medium inwhich the S/W program is stored, or a product traded between a sellerand a purchaser.

For example, the computer program product may include an electronicapparatus or a product (e.g., a downloadable application) in the form ofa S/W program electronically distributed through a manufacturer of theelectronic apparatus or an electronic market (e.g., Google Play™ Storeor App Store™). For the electronic distribution, at least a portion ofthe S/W program may be stored in a storage medium or may be temporarilygenerated. In this case, the storage medium may be a storage medium of aserver in the manufacturer or the electronic market or a relay server.

According to one or more embodiments of the present disclosure, anelectronic apparatus may recognize a user by extracting, even when theelectronic apparatus obtains a user image where a face region notdetected, appearance information or behavior information of the userfrom the user image. Accordingly, restrictions on a location or adirection where a camera that obtains the user image is installedappear.

According to one or more embodiments of the present disclosure, anelectronic apparatus may improve a user's convenience of use byautomatically determining whether to make an update and updating a userrecognition model by using collected appearance information or behaviorinformation of the user.

It should be understood that embodiments described herein should beconsidered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

What is claimed is:
 1. An electronic apparatus comprising: a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is further configured to, by executing the one or more instructions: obtain a first user image of a user, recognize the user from a face region of the first user image by using a first recognition model learned based on face information of a plurality of users, extract a first additional feature information regarding the recognized user from the first user image, obtain a second recognition model by allowing the first recognition model to additionally learn based on the extracted first additional feature information, obtain a second user image of the user, and recognize the user from a person region of the second user image by using the second recognition model.
 2. The electronic apparatus of claim 1, wherein the first additional feature information regarding the user comprises at least one of appearance information or behavior information of the user, and wherein the processor is further configured to, by executing the one or more instructions, detect the person region of the user from the first user image and extract the at least one of the appearance information or the behavior information of the user from the person region.
 3. The electronic apparatus of claim 1, wherein the first recognition model is configured to recognize the user based on face information extracted from the first user image, and wherein the second recognition model is configured to recognize the user based on at least one of face information, appearance information, or behavior information extracted from the second user image.
 4. The electronic apparatus of claim 3, wherein the second recognition model is further configured to: recognize, when the face region is not detected in the second user image, the user by using at least one of the behavior information or the appearance information extracted from the person region, and recognize, when the face region is detected in the second user image, the user by using the face information extracted from the face region.
 5. The electronic apparatus of claim 1, wherein the processor is further configured to, by executing the one or more instructions: evaluate a recognition result of the second recognition model, and update the second recognition model according to a result of the evaluation.
 6. The electronic apparatus of claim 5, wherein the processor is further configured to, by executing the one or more instructions, evaluate the recognition result of the second recognition model by comparing a result of recognizing the user by using a second additional feature information extracted from the second user image and a result of recognizing the user by using the face information extracted from the second user image.
 7. The electronic apparatus of claim 6, wherein the processor is further configured to, by executing the one or more instructions: extract the second additional feature information of the user from the second user image, and update the second recognition model by allowing the second recognition model to additionally learn based on the second additional feature information.
 8. The electronic apparatus of claim 7, wherein the processor is further configured to, by executing the one or more instructions, update the second recognition model by allowing the second recognition model to additionally learn in real time based on the second additional feature information.
 9. The electronic apparatus of claim 7, wherein the processor is further configured to, by executing the one or more instructions: store the second additional feature information, and periodically, when the electronic apparatus is in a preset state, or when a request made by the user is input, update the second recognition model by allowing the second recognition model to additionally learn.
 10. The electronic apparatus of claim 1, further comprising a display configured to display information regarding the recognized user by using the second recognition model.
 11. The electronic apparatus of claim 1, wherein the first recognition model is received from an external server.
 12. An operation method of an electronic apparatus, the operation method comprising: obtaining a first user image of a user; recognizing the user from a face region of the first user image by using a first recognition model learned based on face information of a plurality of users; extracting a first additional feature information regarding the recognized user from the first user image; obtaining a second recognition model by allowing the first recognition model to additionally learn based on the extracted first additional feature information of the user; obtaining a second user image of the user; and recognizing the user from a person region of the second user image by using the second recognition model.
 13. The operation method of claim 12, wherein the first additional feature information regarding the user comprises at least one of appearance information or behavior information of the user, and wherein the extracting of the first additional feature information regarding the recognized user from the first user image comprises detecting the person region of the user from the first user image and extracting the at least one of appearance information or behavior information of the user from the person region.
 14. The operation method of claim 12, wherein the recognizing of the user by using the first recognition model comprises recognizing the user based on face information extracted from the first user image, and wherein the recognizing of the user by using the second recognition model comprises recognizing the user based on at least one of face information, appearance information, or behavior information extracted from the second user image.
 15. The operation method of claim 14, wherein the recognizing of the user by using the second recognition model further comprises: recognizing, when the face region is not detected in the second user image, the user by using at least one of the behavior information or the appearance information extracted from the person region, and recognizing, when the face region is detected in the second user image, the user by using the face information extracted from the face region.
 16. The operation method of claim 12, further comprising: evaluating a recognition result of the second recognition model; and updating the second recognition model according to a result of the evaluating.
 17. The operation method of claim 16, wherein the evaluating of the recognition result of the second recognition model comprises evaluating the recognition result of the second recognition model by comparing a result of recognizing the user by using a second additional feature information extracted from the second user image and a result of recognizing the user by using the face information extracted from the second user image.
 18. The operation method of claim 17, further comprising: extracting the second additional feature information of the user from the second user image, and updating the second recognition model by allowing the second recognition model to additionally learn based on the second additional feature information.
 19. The operation method of claim 18, further comprising: storing the second additional feature information, wherein the allowing of the second recognition model to additionally learn and the updating of the second recognition model comprises allowing the second recognition model to additionally learn and updating the second recognition model periodically, when the electronic apparatus is in a preset state, or when a request made by the user is input.
 20. The operation method of claim 12, the method further comprising displaying information regarding the recognized user by using the second recognition model.
 21. A non-transitory computer-readable recording medium having recorded thereon a program for executing the method of claim 12 on a computer. 